{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#用管线命令处理多个步骤"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "管线命令不经常用，但是很有用。它们可以把多个步骤组合成一个对象执行。这样可以更方便灵活地调节和控制整个模型的配置，而不只是一个一个步骤调节。\n",
    "\n",
    "<!-- TEASER_END -->"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##Getting ready"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "这是我们把多个数据处理步骤组合成一个对象的第一部分。在scikit-learn里称为`pipeline`。这里我们首先通过计算处理缺失值；然后将数据集调整为均值为0，标准差为1的标准形。\n",
    "\n",
    "让我们创建一个有缺失值的数据集，然后再演示`pipeline`的用法："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[ 1.05419595,  1.42287309,  0.02368264, -0.8505244 ],\n",
       "       [ 1.42287309,  5.09704588,         nan, -2.46408728],\n",
       "       [ 0.02368264,  0.03614203,  0.63317494,  0.09792298],\n",
       "       [-0.8505244 , -2.46408728,  0.09792298,  2.04110849]])"
      ]
     },
     "execution_count": 1,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from sklearn import datasets\n",
    "import numpy as np\n",
    "mat = datasets.make_spd_matrix(10)\n",
    "masking_array = np.random.binomial(1, .1, mat.shape).astype(bool)\n",
    "mat[masking_array] = np.nan\n",
    "mat[:4, :4]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##How to do it..."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "如果不用管线命令，我们可能会这样实现："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[ 1.05419595,  1.42287309,  0.02368264, -0.8505244 ],\n",
       "       [ 1.42287309,  5.09704588,  0.09560571, -2.46408728],\n",
       "       [ 0.02368264,  0.03614203,  0.63317494,  0.09792298],\n",
       "       [-0.8505244 , -2.46408728,  0.09792298,  2.04110849]])"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from sklearn import preprocessing\n",
    "impute = preprocessing.Imputer()\n",
    "scaler = preprocessing.StandardScaler()\n",
    "mat_imputed = impute.fit_transform(mat)\n",
    "mat_imputed[:4, :4]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[  1.09907483e+00,   2.62635324e-01,  -3.88958755e-01,\n",
       "         -4.80451718e-01],\n",
       "       [  1.63825210e+00,   2.01707858e+00,  -7.50508486e-17,\n",
       "         -1.80311396e+00],\n",
       "       [ -4.08014393e-01,  -3.99538476e-01,   2.90716556e+00,\n",
       "          2.97005140e-01],\n",
       "       [ -1.68651124e+00,  -1.59341549e+00,   1.25317595e-02,\n",
       "          1.88986410e+00]])"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "mat_imp_and_scaled = scaler.fit_transform(mat_imputed)\n",
    "mat_imp_and_scaled[:4, :4]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "现在我们用`pipeline`来演示："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from sklearn import pipeline\n",
    "pipe = pipeline.Pipeline([('impute', impute), ('scaler', scaler)])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "我们看看`pipe`的内容。和前面介绍一致，管线命令定义了处理步骤："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Pipeline(steps=[('impute', Imputer(axis=0, copy=True, missing_values='NaN', strategy='mean', verbose=0)), ('scaler', StandardScaler(copy=True, with_mean=True, with_std=True))])"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pipe"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "然后在调用`pipe`的`fit_transform`方法，就可以把多个步骤组合成一个对象了："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[  1.09907483e+00,   2.62635324e-01,  -3.88958755e-01,\n",
       "         -4.80451718e-01],\n",
       "       [  1.63825210e+00,   2.01707858e+00,  -7.50508486e-17,\n",
       "         -1.80311396e+00],\n",
       "       [ -4.08014393e-01,  -3.99538476e-01,   2.90716556e+00,\n",
       "          2.97005140e-01],\n",
       "       [ -1.68651124e+00,  -1.59341549e+00,   1.25317595e-02,\n",
       "          1.88986410e+00]])"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "new_mat = pipe.fit_transform(mat)\n",
    "new_mat[:4, :4]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "可以用Numpy验证一下结果："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "True"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "np.array_equal(new_mat, mat_imp_and_scaled)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "完全正确！本书后面的主题中，我们会进一步展示管线命令的威力。不仅可以用于预处理步骤中，在降维、算法拟合中也可以很方便的使用。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##How it works..."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "前面曾经提到过，每个scikit-learn的算法接口都类似。`pipeline`最重要的函数也不外乎下面三个：\n",
    "\n",
    "- `fit`\n",
    "- `transform`\n",
    "- `fit_transform`\n",
    "\n",
    "具体来说，如果管线命令有`N`个对象，前`N-1`个对象必须实现`fit`和`transform`，第`N`个对象至少实现`fit`。否则就会出现错误。\n",
    "\n",
    "如果这些条件满足，管线命令就会运行，但是不一定每个方法都可以。例如，`pipe`有个`inverse_transform`方法就是这样。因为由于计算步骤没有`inverse_transform`方法，一运行就有错误："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "ename": "AttributeError",
     "evalue": "'Imputer' object has no attribute 'inverse_transform'",
     "output_type": "error",
     "traceback": [
      "\u001b[1;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[1;31mAttributeError\u001b[0m                            Traceback (most recent call last)",
      "\u001b[1;32m<ipython-input-12-62edd2667cae>\u001b[0m in \u001b[0;36m<module>\u001b[1;34m()\u001b[0m\n\u001b[1;32m----> 1\u001b[1;33m \u001b[0mpipe\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0minverse_transform\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mnew_mat\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m",
      "\u001b[1;32md:\\programfiles\\Miniconda3\\lib\\site-packages\\sklearn\\utils\\metaestimators.py\u001b[0m in \u001b[0;36m<lambda>\u001b[1;34m(*args, **kwargs)\u001b[0m\n\u001b[0;32m     35\u001b[0m             \u001b[0mself\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mget_attribute\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mobj\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m     36\u001b[0m         \u001b[1;31m# lambda, but not partial, allows help() to work with update_wrapper\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m---> 37\u001b[1;33m         \u001b[0mout\u001b[0m \u001b[1;33m=\u001b[0m \u001b[1;32mlambda\u001b[0m \u001b[1;33m*\u001b[0m\u001b[0margs\u001b[0m\u001b[1;33m,\u001b[0m \u001b[1;33m**\u001b[0m\u001b[0mkwargs\u001b[0m\u001b[1;33m:\u001b[0m \u001b[0mself\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mfn\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mobj\u001b[0m\u001b[1;33m,\u001b[0m \u001b[1;33m*\u001b[0m\u001b[0margs\u001b[0m\u001b[1;33m,\u001b[0m \u001b[1;33m**\u001b[0m\u001b[0mkwargs\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m\u001b[0;32m     38\u001b[0m         \u001b[1;31m# update the docstring of the returned function\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m     39\u001b[0m         \u001b[0mupdate_wrapper\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mout\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mself\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mfn\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n",
      "\u001b[1;32md:\\programfiles\\Miniconda3\\lib\\site-packages\\sklearn\\pipeline.py\u001b[0m in \u001b[0;36minverse_transform\u001b[1;34m(self, X)\u001b[0m\n\u001b[0;32m    265\u001b[0m         \u001b[0mXt\u001b[0m \u001b[1;33m=\u001b[0m \u001b[0mX\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m    266\u001b[0m         \u001b[1;32mfor\u001b[0m \u001b[0mname\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mstep\u001b[0m \u001b[1;32min\u001b[0m \u001b[0mself\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0msteps\u001b[0m\u001b[1;33m[\u001b[0m\u001b[1;33m:\u001b[0m\u001b[1;33m:\u001b[0m\u001b[1;33m-\u001b[0m\u001b[1;36m1\u001b[0m\u001b[1;33m]\u001b[0m\u001b[1;33m:\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m--> 267\u001b[1;33m             \u001b[0mXt\u001b[0m \u001b[1;33m=\u001b[0m \u001b[0mstep\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0minverse_transform\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mXt\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m\u001b[0;32m    268\u001b[0m         \u001b[1;32mreturn\u001b[0m \u001b[0mXt\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m    269\u001b[0m \u001b[1;33m\u001b[0m\u001b[0m\n",
      "\u001b[1;31mAttributeError\u001b[0m: 'Imputer' object has no attribute 'inverse_transform'"
     ]
    }
   ],
   "source": [
    "pipe.inverse_transform(new_mat)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "但是，`scalar`对象可以正常运行："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "collapsed": false,
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[ 1.05419595,  1.42287309,  0.02368264, -0.8505244 ],\n",
       "       [ 1.42287309,  5.09704588,  0.09560571, -2.46408728],\n",
       "       [ 0.02368264,  0.03614203,  0.63317494,  0.09792298],\n",
       "       [-0.8505244 , -2.46408728,  0.09792298,  2.04110849]])"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "scaler.inverse_transform(new_mat)[:4, :4]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "只要把管线命令设置好，它就会如愿运行。它就是一组`for`循环，对每个步骤执行`fit`和`transform`，然后把结果传递到下一个变换操作中。\n",
    "\n",
    "使用管线命令的理由主要有两点：\n",
    "\n",
    "- 首先是方便。代码会简洁一些，不需要重复调用`fit`和`transform`。\n",
    "- 其次，也是更重要的作用，就是使用交叉验证。模型可以变得很复杂。如果管线命令中的一个步骤调整了参数，那么它们必然需要重新测试；测试一个步骤参数的代码管理成本是很低的。但是，如果测试5个步骤的全部参数会变都很复杂。管线命令可以缓解这些负担。"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.5.1"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}
